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Cross-Reference to Related Applications 
[0001] This application claims the benefit of the following provisional patent 
applications, each of which is incorporated by reference in its entirety: U.S. Provisional 
Application No. 60/506,206, filed September 25, 2003 entitled "Printer Driver, Interface and 
Method for Selecting and Printing Representations of Audio, Video or Processed Information;" 
U.S. Provisional Application No. 60/506,263, filed September 25, 2003 entitled "Systems and 
Methods for Printing Audio or Video Paper;" U.S. Provisional Application No. 60/506,302, filed 
September 25, 2003 entitled "Printer Including Interface and Specialized Information Processing 
Capabilities;" U.S. Provisional Application No. 60/506,303, filed September 25, 2003 entitled 
"Printer Including One or More Specialized Hardware Devices;" and U.S. Provisional 
Application No. 60/506,41 1, filed September 25, 2003 entitled "Printer for Non-Paper and 
Multiple Media Types." 

[0002] This application is also related to the following co-pending patent 
applications, each of which is incorporated by reference: U.S. patent application entitled, "Printer 
Having Embedded Functionality for Printing Time-Based Media," filed March 30, 2004, 
Attorney Docket No. 20412-08430; and U.S. patent application entitled, "Networked Printing 
System Having Embedded Functionality for Printing Time-Based Media," filed March 30, 2004, 
Attorney Docket No. 20412- 08431. This application is also related to the following co-pending 
patent applications (hereafter the "Media Content Processing Applications"), each of which is 
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incorporated by reference U.S. patent application entitled, "Printer with Audio/Video 
Localization," filed March 30, 2004, Attorney Docket No. 20412-08356; U.S. patent application 
entitled, "Music Processing Printer," filed March 30, 2004, Attorney Docket No. 20412-08358; 
U.S. patent application entitled, "Printing System With Embedded Audio/Video Content 
Recognition and Processing," filed March 30, 2004, Attorney Docket No. 20412-08394. 

[0003] This application further incorporates by reference in their entirety the 
following co-pending patent applications (hereinafter, "the Video Paper patent applications"): 
U.S. patent application entitled, "Printable Representations for Time-based Media," filed March 
30, 2004, Attorney Docket No. 20412-08497; U.S. Application No. 10/001,895, filed November 
19, 2001; U.S. Application No. 10/001,849, filed November 19, 2001; U.S. Application No. 
10/001,893, filed November 19, 2001; U.S. Application No. 10/001,894, filed November 19, 
2001; U.S. Application No. 10/001,891, filed November 19, 2001; U.S. Application No. 
10/175,540, filed June 18, 2002; and U.S. Application No. 10/645,821, filed August 20, 2003. 

Field of Invention 

[0004] The invention generally relates to printers that have embedded functionality 
for printing multimedia data. 

Background 

[0005] A printer in a computer system typically is seen as handling the mechanics 
of physically outputting information on a medium, typically paper. The printer is somewhat of a 
passive participant in the computer system. Printers handle a bit of higher level control for 
controlling printing such as keeping a queue of documents to be printed, indicating the toner is 
low, or providing an out-of-paper message. Printers do not typically provide any assistance with 
processing actual content for printing on an output medium. 
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[0006] Knowledge workers are now making increasing use of multimedia data, 
particularly in reports and presentations. The multimedia data typically includes time based data 
sometimes also referred to as time based media, examples of which include audio clips, 
animations, and video clips. A computer, typically a desktop computer or a laptop, connected to 
the printer via a network interface or a direct connection, has to have the necessary hardware 
and/or software for modifying the multimedia content for printing, formatting that content for a 
desired medium, and selecting the desired medium upon which to imprint the content. If users at 
different computers in a network desire to perform multimedia content processing functions and v 
output to different media, the different computers each have to have the appropriate hardware 
and /or software. Furthermore, their computers have to handle the computational load required 
for multimedia content processing and for supporting specific formats for storing and printing 
multimedia content. 

[0007] It is desirable to offload at least part of the computational load to an 
intelligent printer that can perform multimedia content processing, particularly for time-based 
data, such as editing and content formatting independently from a computer to which it is 
connected. It is desirable also that the printer provides a variety of output media options. It is 
also desirable that the printer can provide storage for the various forms of output media for 
example in a multimedia content database. A printer that can independently maintain a 
multimedia database provides streamlined storage of the multimedia content. 

Summary of the Invention 

[0008] In one aspect, the present invention provides a printer with an embedded 
multimedia server. One embodiment of the printer with an embedded multimedia server 
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comprises a chassis for housing a print engine, a multimedia server, a non-volatile storage device 
for storing a database of multimedia content, and a user input device accessible on the printer 
chassis. In this embodiment, the print engine controls printing to a plurality of media storage 
forms, including removable storage media forms. The print engine is coupled to at least one 
media holder and is communicatively coupled to the multimedia server. The user interface 
device is communicatively coupled to the multimedia server. The multimedia server is 
communicatively coupled to at least one media content interface through which it receives 
multimedia content from one or more media content sources. This embodiment can further 
comprise a display accessible on the printer chassis and communicatively coupled to the 
multimedia server for displaying data under the control of the server. 

[0009] In one instance, the multimedia server also includes a multimedia database 
manager which is communicatively coupled to the non- volatile storage medium storing the 
database. In another instance, there is at least one network interface communicatively coupled to 
the multimedia server which includes a web server. In another example, the multimedia server 
further comprises a web browser which displays data on the display accessible on the printer 
chassis. 

[0010] In another aspect, the present invention provides a method for processing 
multimedia content for use in a printer with an embedded multimedia server. One embodiment 
of such a method in accordance with the present invention comprises performing multimedia 
content processing, and outputting a content selection based on criteria. Another embodiment of 
a method in accordance with the present invention further comprises generating an index 
document for the content selection based on criteria. Another method embodiment further 
comprises monitoring time based multimedia content based on criteria. 
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Brief Description of the Drawin2S 

[0011] Figure 1 illustrates a printer with an embedded multimedia server in 
accordance with an embodiment of the present invention. 

[0012] Figure 2 illustrates a functional software architecture for a multimedia 
server embedded in a printer in accordance with an embodiment of the present invention. 

[0013] Figure 3 illustrates a method for processing multimedia content selections 
for use by a multimedia server embedded in a printer in accordance with an embodiment of the 
present invention. 

[0014] Figure 4 illustrates a method for printing a multimedia content selection 
based on criteria for use by a multimedia server embedded in a printer in accordance with an 
embodiment of the present invention. 

[0015] Figure 5 illustrates a method for monitoring time based data content for use 
in a printer with an embedded multimedia server in accordance with an embodiment of the 
present invention. 

[0016] Figure 6 illustrates a method for editing time based data content for use in a 
printer with an embedded multimedia server in accordance with an embodiment of the present 
invention. 

[0017] Figure 7 illustrates an example of a method for printing a video content 
selection based on user defined criteria wherein the criteria is that each clip indicates a time 
period in which a different person begins speaking for use in a printer with an embedded 
multimedia server in accordance with an embodiment of the present invention. 

[0018] Figure 8 illustrates an example of a method for printing a video content 
selection based on user defined criteria wherein the criteria is that each clip indicates a time 
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period in which a person was speaking in the location for use in a printer with an embedded 
multimedia server in accordance with an embodiment of the present invention. 

[0019] Figure 9 illustrates an example of a method for printing a video content 
selection based on user defined criteria wherein the criteria is that each clip indicates a time 
period in which a participant is present in a video selection for use in a printer with an embedded 
multimedia server in accordance with an embodiment of the present invention. 

[0020] Figure 10 illustrates an example of a method for printing a video content 
selection based on user defined criteria wherein the criteria is that each clip indicates a time 
period in which a person was speaking in a video for use in a printer with an embedded 
multimedia server in accordance with an embodiment of the present invention. 

[0021] Figure 1 1 illustrates an example of a method for capturing referenced 
multimedia content in a content selection designated for printing for use in a printer with an 
embedded multimedia server in accordance with an embodiment of the present invention. 

[0022] Figure 12 illustrates an example of a method for determining whether a 
printed content selection has been removed from an output holder for use in a printer with an 
embedded multimedia server in accordance with an embodiment of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
[0023] Figure 1 illustrates a printer 100 with an embedded multimedia server 104 in 
accordance with an embodiment of the present invention. The printer 100 comprises a printer 
chassis 102 that provides a housing for various components such as a print engine 140, a 
motherboard 106 comprising a multimedia server 104 and a local memory 108, a non- volatile 
storage device 114 illustrated in this example as a memory hard disk for storing a database 1 16 
of multimedia content data, at least one user input device 152 accessible on the printer chassis 
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102, a display 148 accessible on the printer chassis 102, at least one media content interface 136 
accessible through the chassis 102, a speaker 150 accessible on the printer chassis 102, and at 
least one removable storage medium holder 142 that can be detached from the chassis 102. 

[0024] The print engine 140 is communicatively coupled to the multimedia server 
104 on the motherboard 106 in this embodiment. The print engine 140 includes a processor 166, 
which can be a general purpose computer processor, a local memory 164, and at least one media 
form reader/ writer 156, examples of which include a DVD reader / writer, a CD reader / writer, 
a SD reader / writer, an audio cassette reader / writer, a videocassette reader / writer, a floppy 
disk reader / writer, a ZIP disk reader / writer, a flash card reader / writer, a micro-drive reader / 
writer, and a memory stick reader / writer. Other examples of media forms that may be read or 
written to are paper, plastic sheets, magnetic strips, or any other suitable type of removable 
storage media devices. In this embodiment, selected content to be printed is downloaded to the 
local memory 164 by the multimedia server 104 to the print engine 140. In another embodiment, 
the multimedia server 104 can be accessed by a media content interface 136 for transferring 
content data. For example, the media content interface 136 can include a broadband network 
interface through which an external computing device 154 such as a laptop or personal digital 
assistant (PDA) receives or sends time-based media data. In this example, the at least one 
removable storage medium holder 142 is controlled by the print engine 140 for the printing 
processing. For example, the print engine 140 controls the positioning of a medium for reading 
and writing. Examples of a removable media holder 142 are a tray or bandolier configured for 
holding one or more forms of removable storage media. 

[0025] The display 148 accessible on the printer chassis is communicatively 
coupled to the multimedia server 104 for displaying data under its control. In another 
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embodiment, the display 148 is also communicatively coupled to the p>rint engine 140 for 
displaying data under its control as well. The user input device 152 is communicatively coupled 
to the multimedia server 104 for receiving input which can indicate user preferences and criteria 
for content based processing. The input device 152 can include one or more of the following 
examples: a pointing device, buttons, a touch screen, a speech recognition system with a 
microphone, a keypad, or a keyboard or any other suitable input device. In another example, the 
device 152 is also communicatively coupled to the print engine 140 to receive user input directed 
to controlling the printing process such as the number of copies. In another example, the 
multimedia server 104 can handle all user input. 

[0026] The multimedia server 104 can be implemented in hardware, software, 
firmware or a combination of both. Preferably, software associated with multimedia server 104 
is executed on a separate processor from the processor 166 within the print engine 140. Again, a 
general purpose computer processor can be used. The multimedia server 104 performs 
multimedia content processing, and manages the database 1 16 of multimedia content. The 
multimedia server 104 is communicatively coupled to receive content data for different types of 
media in different formats, particularly time-based multimedia content data, from media content 
sources 154 via at least one media content interface 136. Additionally, the multimedia server 
104 can send content data for different types of media in different formats, particularly time- 
based multimedia content data, to media content recipients 154 via at least one media content 
interface 136. Preferably, the media content interface 136 includes a broadband network 
interface. One example of multimedia content processing that the server 104 can perform is 
receiving content data in one format, and converting it to another format for output on a 
removable storage medium, storage in another form in the database 1 16, or for relaying over a 
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media content interface 136, for example, a network interface to another storage location. For 
example, if the printer 100 receives information from a Bluetooth device, like an Anoto pen, the 
server 104 saves the data in a portable document format (PDF) in the database 1 16 or relays it as 
a PDF over a network interface 136 to another storage location for downloading later on the 
user's home computer. Such a relay function could be very useful for Bluetooth video cameras, 
cell phones with cameras that also have Bluetooth interfaces, and any other Bluetooth-enabled 
device. Relaying of data can be done for a batch re-transmission of stored recorded data, as 
controlled by a touch panel interface 152 on the printer, or for a live re-transmission as data is 
being recorded. 

[0027] In the illustrated embodiment, the multimedia server 104 comprises a media 
content presenter 158, examples of which include software capable of displaying fixed content 
such as text, a photo, or a diagram on the display 148, software and/or hardware for playing 
video data on the display 148 or software and/or hardware for playing audio data through the 
speaker 150. 

[0028] The media content interface 136 can be designed to accommodate various 
formats for multimedia data including time-based data. Because of the great variety of types and 
formats of media data, the media content interface 136 may take any number of forms. The 
following are some illustrative examples of different media content interfaces 136. In particular 
implementations, the printer 100 may have only one or only a subset of these types of interfaces. 
One example is a network interface for receiving data over a network, for example a computer 
network (e.g. internal company network or the Internet, or a computer connection relaying data 
originating from other content capture sources such as cellular telephones, a TV receiver, or a 
satellite, or a GPS receiver.) The interface may comprise a port for connecting a device using a 
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connection type such as USB, serial, Fire Wire, SCSI, IDE, RJ1 1, optical, composite video, 
component video, or S-video or any other suitable connection type. A media content interface 
can also include a frame grabber for accessing data from a composite video source. In another 
example, a transcoder may also be used for accessing data from a composite video source. In 
one example, the transcoder is implemented in hardware that converts an MPEG-2 file to a 
streamable format in real-time. An example of such hardware is the Via Technologies MPEG 
encoding chip provided in the Abocom HA2500 Multimedia Gateway (www.abocom.com). The 
same transcoding function could be provided in software. Another example of an interface 136, 
includes an NTSC converter for converting to digital form data from a VGA port. Another 
example of an interface 136 includes an analog to digital converter for transforming analog voice 
signals from a microphone to a digital format, for example a streaming format. An analog to 
digital converter could also be used to convert an image directly from a video recorder into a 
digital format. In another example, the interface 136 may comprise a media processing system 
for converting, encoding and compressing raw video data. 

[0029] In another example, the interface 136 may comprise a USB port for 
receiving data from a pen-based data capture system and a pen capture module. In one 
embodiment, the pen capture module includes ultrasonic pen capture hardware and the 
multimedia server 104 includes logic to interface with the ultrasonic pen capture system. An 
example of an ultrasonic pen capture system is one provided by InkLink and described at 
http://www.siibusinessproducts.com/products/link-ir-p.html. 

[0030] Another embodiment of the interface 136 may include one or more wireless 
communication interfaces, examples of which may include RF interface(s) such as Bluetooth, 
802.11 standards, 802.15 standards, 802.16 standards, satellite communications, and cellular 
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network communications such as 3G and 4G and GPRS. Another wireless example is an 
infrared (IR) interface. 

[0031] For more examples of media content interfaces, see U.S. patent application 
entitled "Printer Having Embedded Functionality for Printing Time-Based Media," filed March 
30, 2004 Attorney Docket No. 20412-08340 which has been incorporated by reference in its 
entirety. 

[0032] Figure 1 further illustrates that optionally, the printer 100 can include one or 
more embedded data capture modules 168. Examples of data capture devices include an event 
sensor that receives a signal about or otherwise observes a real- world event (e.g., a motion 
detector or a sound detector), a video recorder, a digital camera, a film based camera, a 
microphone, or a signal receiver. Examples of a signal receiver are a cellular telephone receiver, 
a satellite receiver, or a TV receiver. A data capture module 168 can be communicatively 
coupled to a media content interface 136 for any processing of its output data, for example 
converting, encoding or compressing, placing its output data in a form suitable for storage in the 
multimedia content database 1 16. A data capture module 168 can also be communicatively 
coupled to the multimedia server 104 if it outputs data in a suitable form for storage in the 
database 116. 

[0033] For more examples of embedded data capture devices, see U.S. patent 
application entitled "Printer Having Embedded Functionality for Printing Time-Based Media," 
filed March 30, 2004 Attorney Docket No. 20412-08340 which has been incorporated by 
reference in its entirety. 

[0034] Figure 2 illustrates a functional software architecture for a multimedia 
server 104 embedded in a printer in accordance with an embodiment of the present invention. 
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Various examples of software portions for performing examples of content processing functions 
are illustrated. In particular implementations, the multimedia server 1 04 may have only one or 
only a subset of these types of software portions. The multimedia server 104 comprises a control 
unit 214 software portion including an event handler 232, the control unit 214 being 
communicatively coupled to each software portion including, applications software portions 202, 
a database manager 204, a content indexing software portion 208, a user interface software 
portion 210, and a network interface manager illustrated here as a web server 1 10, a streaming 
server 1 12 including a streaming data player 158, a web browser 160, a content editing portion 
218, scheduling software 220, multimedia correlation software 222, a multimedia format 
transformation software portion 224, a multimedia content searching software portion 226, a 
multimedia content recognition software portion 228, and a multimedia compression software 
portion 230. For purposes of discussion, each of these software portions can either directly or 
indirectly be invoked by or invoke any other software portion. For example, the scheduling 
software portion 220 can call the streaming server 1 12, or a specific application 202 can invoke 
the multimedia content editing software 218 to perform a specific task such as video clip 
segmentation based on input parameters. 

[0035] The control unit 214 software portion provides a centralized management 
point for the multimedia server 104. Its event handler 232 can be implemented in one example 
as an application programming interface (API) which receives notifications (e.g. in the form of 
messages) from the other software portions or interfaces indicating the occurrence of an event. 
For example, the user interface software 210 processes input from the user input device 152 or 
input from display 148, which may include a touch screen or have a pointing device such as a 
mouse or stylus attached to it. The user interface software 210 forwards the input to the event 
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handler 232 which forwards it to the software portion(s) to which it applies. Furthermore, the 
user interface software 210, updates the display in accordance instructions or data received from 
another software portion, for example, the web browser 160 in the web server 1 10. In another 
example, the scheduling software 220 notifies the event handler 232 that a designated time for 
printing a content selection has been reached or soon will be. The event handler 232 notifies the 
database manager 204 which in turn retrieves the content selection into local memory 108, and 
the control unit 214 sends a message to the print engine 140 which in turn prints the content 
selection on the designated medium at the designated time. 

[0036] The database manager 204 stores and retrieves content selections as well as 
creates meta-data associated with each content selection to identify it or maintain instructions 
associated with it. An example of a content selection is a video file, an audio file, or a text file. 

[0037] The application(s) 202 can be user created applications. 

[0038] The content indexing portion 208 generates a content index document that 
can be printed as a paper document. The index document can also be printed on other tangible 
media such as transparency sheets for overhead projectors, film, slides, canvass, glass, stickers, 
or any other medium that accepts such markings. A paper representation is easy to browse by a 
user. The information contained in the content index document can take various forms based on 
criteria established by a user or an application. Examples of information that can be represented 
in a content index document are an inventory of one or more categories of content selections 
stored in the database 1 16 or in the local memory 108 for the multimedia server 104, detected 
events, a time line of events, the location of key data in the content or information about a 
transaction (e.g. a relay transmission between a source and a recipient) regarding the content. In 
one example, an index document includes portions of multimedia content such as audio clips or 
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video clips printed on the paper. Other identifying information can also be included examples of 
which are titles, names of participants, times, event detected (e.g. door opened.), or size of 
content selection or portion. In another example, a bar code is included for each clip indicating 
to a multimedia player reading the content selection the location of the clip on the medium. In 
one example, bar codes can be provided that fully encode the digital contents of the full 
resolution key frame. In another example, the bar codes can provide a link to a version stored 
off-line. A paper document with video images is often called a video paper document. A paper 
document with audio images is often called an audio paper document. Various examples of 
index documents, in particular audio paper and video paper are illustrated in the Video Paper 
patent applications. 

[0039] The web server 1 10 provides content to a web browser 1 60, or any other 
remote web browser across a network and access for hosts on the World Wide Web or the 
Internet for the multimedia server 104. A user may access external websites by indicating 
selections via the display 148 or the input device 152 in conjunction with the display 148 which 
may be forwarded to the server 1 10 or to some other server of the Internet. 

[0040] The streaming server 1 12 interacts with digital data to be input to format it 
in a streaming format for display under the control of the player 158. The streaming server 1 12 
can also perform tasks such as monitoring streamed data based on criteria. Examples of criteria 
include event detection or the elapsing of a time period. 

[0041] The multimedia content editing portion 218 performs tasks such as 
segmenting a clip from an audio or a video content selection as well as text editing capabilities or 
adding sounds or images to a content selection. 
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[0042] The multimedia correlation portion 222 performs tasks such as correlating 
video content to audio content based on a criteria such as a time period, a location, a participant 
or other criteria. 

[0043] The multimedia format transformation portion 224 performs tasks related to 
transforming content in one format to another format. For example, a portion of text can be 
converted from html to speech in a streaming audio format or voice XML. Similarly, data from 
a satellite transmission can be represented on a streaming format for printing to a DVD. 

[0044] The multimedia compression portion 230 performs services such as 
compressing content for storage in the database or printing on a removable storage medium (SD) 
and subsequently decompressing content for playback on the player 1 58 for example. 

[0045] The multimedia content recognition software portion 228 performs 
recognition functions for various forms of media. For example, for audio content, it can apply 
voice matching methods such as voice signatures, and speech recognition methods. For 
handwritten text, it can include handwriting recognition methods. For video, it may, for 
example, employ scene or face recognition methods. 

[0046] The multimedia content searching software portion 226 performs searching 
functions for multimedia content. For example, it can perform text searching. Similarly, voice 
signatures can be searched as well as recognized sounds or faces or other recognized visual 
images. 

[0047] The following figures illustrate embodiments of methods for multimedia 
content processing by an embedded multimedia server in a printer in accordance with the present 
invention. For illustrative purposes only, the method embodiments illustrated are discussed in 
the context of the system embodiments of Figure 1 and/or Figure 2. 
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[0048] Figure 3 illustrates a computer-implemented method for processing 
multimedia content selections for use by a multimedia server embedded in a printer in 
accordance with an embodiment of the present invention. The multimedia server 104 performs 
302 multimedia content processing and generates 306 an index document for selected content 
based on criteria. The multimedia server 104 causes the outputting 308 of a content selection 
based on criteria. The multimedia server can output 308 a content selection to a device attached 
via a network such as an external computer 154 or other communication interface such as the 
speaker 150 in addition to directing the print engine to output the content selection on a 
removable medium. For example, the multimedia server 104 stores the data provided by the 
ultrasonic pen system and saves it as a content selection in the database 1 16. From the display, 
the user can select to print the information or email it. The multimedia correlation portion 222 
links the content selection of the handwriting to other multimedia content including events 
occurring nearby the printer at the time the user was using the pen system to create the 
handwriting. For example, the audio or video of others can show that there were two witnesses 
to a signature. A printout of a generated index document includes barcodes that provide links to 
this other multimedia recorded at the same time. Packaging this capability in a printer would be 
useful since it avoids the need to provide a separate PC with this functionality. 

[0049] Figure 4 illustrates a method for a multimedia content selection based on 
criteria for use by a multimedia server embedded in a printer in accordance with an embodiment 
of the present invention. The user interface software portion 210 displays 402 a list of content 
selections on an embedded printer display. Consider an example in which the multimedia 
database 116 includes a music catalog, or a video database, or a movie database or a digital 
photo catalog, or a combination of one or more of these media types. The user interface software 
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210 displays a print dialog box on the display 148 which presents the user with a list of the 
songs, a list of video recordings, a list of movies or a list of photos, resident in the printer 100. 
The user can choose to "print" any of those songs, videos, movies or photos. Responsive to 
input indicating 404 a content selection, if previewing is a feature of the printer, optionally the 
user interface software 210 displays 406 an indicator for previewing the content selected. 
Responsive to user input indicating 408 previewing is requested, the content selection is 
provided 414 in the sensual form appropriate to its type. For example, if a song is selected and 
previewing requested, the event handler 232 sends a message to the streaming server 1 12 
requesting that the song be played. The streaming server 1 12 requests the song from the 
database manager 204 which retrieves it from the database 1 16 where it is stored in this example 
in a streamed format. Upon receipt, the streaming player 158 streams the song in real-time to a 
digital to analog converter, the analog output of which is directed to the speaker 150 through 
which the user hears the song. In a similar fashion, the user interface software 210 displays on 
display 148 the selected video or movie streamed from the player 158 in real-time, or the 
selected photo forwarded from the web server 1 10 or a photo processing application 202. During 
the previewing of the content selection, a user may have made edits, such as marks or 
annotations, to the content selection using multimedia content editing software 218 which can be 
saved to a new version of the content selection temporarily stored in the local memory 108. 
Other examples of content processing that a user can perform are described in the Media Content 
Processing applications. Responsive to user input indicating 413 not to proceed with printing, no 
further action is taken 415 to print the selection. Responsive to user input indicating 413 that 
outputting is to proceed, the content indexing software portion 208 generates 418 an index 
document for the content selection. The index paper document can indicate only the marked 
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portions or the whole content selection (See Video Paper patent applications). The print out 
would be a convenient time-saver allowing one to browse what occurred in a meeting, while 
away from a network connection that would enable replaying of the digital media. The print out 
would also be a convenient time-saver allowing one to browse what occurred in a movie, while 
away from a network connection that would enable replaying of the digital media. 

[0050] Responsive to user input indicating 410 a preferred print medium, the 
multimedia server 104 transfers the content selection in a format suitable for that print medium 
to the print engine (e.g. to local memory 164) which prints 416 the content selection on the 
indicated preferred medium. If user input does not indicate a preferred medium, the multimedia 
server 104 transfers the content selection in a format suitable for the default print medium to the 
print engine which prints 412 the content selection on the default medium suitable for the type of 
content. In the case of the song catalog, examples of the print medium that can be used are a CD 
or cassette. In the case of a video or a movie, examples of the print medium that can be used are 
a DVD or videocassette. In the case of the photos, examples of the print medium that can be 
used are paper or a memory stick. In one example, the database of content selections can be 
updated through a network interface 136 by downloading from a web site via the web server 1 10 
for instance. These examples illustrate that this printer could be very useful in a walk-up kiosk 
as can be found in a convenience store. In another example, the database is communicatively 
coupled to an internal database (e.g. of company recorded meetings). Additionally, in the 
example of an embedded digital photo catalog, a user can select pictures for printing or generate 
an index document of photos without having to turn on a separate computer networked to the 
printer. For example, if every time a user prints a jpeg image, the printer automatically or under 
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control of the print dialog box, saves a copy of the jpeg on the printer, she can build a personal 
photo archive and peruse the archive without turning on her desktop computer. 

[0051] Figure 5 illustrates a method for monitoring time based data content for use 
in a printer with an embedded multimedia server in accordance with an embodiment of the 
present invention. Monitoring software, for example, the streaming server 1 12 monitors 502 
content from a time-based media source. The content indexing software portion 208 
automatically generates 504 an index document illustrating a log for the content being monitored 
based on criteria. Criteria can be user defined or generated by an application specific software 
portion and processed by the monitoring software such as the streaming server 112. Consider, 
for example, a printer with an embedded streaming video monitoring capability such as can be 
provided by the streaming server 1 12. A user configures the printer to monitor a streaming video 
source so that it constructs a storable representation for the streamed data, perhaps playing it 
through a video card (e.g. 158) and recording the analog output (e.g. using a video recorder 168) 
to generate a new digital representation (e.g., using an analog digital converter as part of an 
interface 136), if necessary. A printout is generated that shows a log of the recorded data, 
perhaps including bar codes that fully encode the digital contents or provide a link to its off-line 
storage location. Consider another example of a printer with multi-channel streaming video 
monitoring capability. A user configures the printer, which in this example has multiple 
processors and a wide bandwidth Internet link to monitor more than one streaming video source 
simultaneously so that the printer constructs a storable representation for the streamed data, 
perhaps playing it through more than one video card and recording the analog outputs to generate 
new digital representations, if necessary. A printout is generated that shows a log of the recorded 
data from each of the channels, perhaps including bar codes that fully encode the digital contents 
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or provide a link to its off-line storage location. Another example illustrates an embodiment 
wherein the monitored time based data is streamed audio. In this example, a user configures a 
printer to monitor a streaming audio site with intermittent broadcasts, such as a surveillance 
system or scanner. An event sensor may detect a sound of which a monitoring application 202 is 
notified, and voice detection software that can be part of the multimedia content recognition 
software 228, can used to distinguish types of sound, for example, a female human voice. 
Content editing software 218 such as segmentation software is applied to generate 504 a paper 
index document that shows when people spoke. If voice recognition software 208 is used, its 
identification results can be indicated on the paper document as well. In one example, the paper 
document is just a printed activity summary and does not include a means for accessing the 
digital version of the recording, such as bar codes. In another example, bar codes can be 
included as well. In another example, the printer can also print the paper document on a 
removable storage medium that contains the digital data. The bar codes identify to a media 
player in which the removable medium is inserted and the location of the segments when people 
spoke. 

[0052] In another embodiment, the multimedia server 104 acts to monitor 
interactions with an external computer such as a desktop PC to which it is connected. For 
example, the video cable from the PC goes to the printer 100 first where the signal is split with 
one copy going to the PC and another copy to screen capture hardware 136 (e.g. VGA). A PC 
interaction monitoring application 202 does a differencing between successive frames and saves 
frames with a difference that exceeds a threshold on secondary storage 1 14 in the printer. This 
offloads such processing from the PC, thereby improving responsiveness and user experience, 
and provides an easily browsable record of a user's activities during the day. The user can 
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choose to print selected frames, hence the relationship to the printer. Print outs can be generated 
on demand by pressing a button on the printer's console (e.g. 152), a dialog box on the client PC 
154 or on the printer's display 148, or at times indicated by scheduling software 220. This 
printer 100 can be set to produce video files by selecting a frame every predetermined number of 
seconds irrespective of the content difference. This lets a user watch a replay of what he did. It 
can also be effectively compressed by the multimedia compression software 230 since the 
differences between frames are minor. Also, the video can be edited with video editing software 
218. Video paper can be generated from the video files. 

[0053] In another example, user interaction with a keyboard can be determined. In 
one example, assuming the user has a wireless keyboard and the printer has a wireless keyboard 
interface 136. A hardware tap into the signals emitted by the keyboard is provided. This tap 
records everything a user types. Each key click is recorded together with the time stamp when it 
was received. A timeline representation for a user's activities is constructed that can be printed 
out or displayed. The display is controlled by a parameter that gives a time gap figure. That is, 
key clicks less than the threshold are displayed adjacent to one another. Exceeding the threshold 
inserts a carriage return in the display. Thus if a user types a normal document, the timeline 
would look similar to that document. But if they frequently stop for phone calls or chatting, the 
gaps in the time line would indicate that. This display would be a productivity-monitoring tool. 
It would also be a useful index that could be searched with a full text search engine and would 
help answer questions like "when did I type this paragraph?" or "when did I revise it?" The time 
information could be correlated with information from other sources, such as the VGA screen 
capture, to see what the user had displayed on their screen when they typed or revised that 
paragraph. 
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[0054] In another example, additional software could be installed on the user's PC 
154 that transmits via a network interface 136 or another wired or wireless interface 136 
identifiers for files opened, applications invoked, URL's browsed and the names and locations of 
files sent to the printer. The content indexing software 208 or the PC interaction application 202 
can create a document displaying this information that can be displayed or printed out. 

[0055] In another example, the server 104 has a video camera interface 136 such as 
a frame grabber that receives composite video signals. The camera monitors activities nearby 
the printer and saves key frames or video clips in a file on an internal buffer store 108 when 
activity exceeds a threshold. This threshold can be defined by the difference between successive 
key frames or by a measure applied to clips extracted from a video stream. For example a clip 
length parameter (e.g., 30 seconds) is defined. A separate clip is saved automatically every 30 
seconds by the video capture hardware 136. That file is closed and the clip capture proceeds. A 
signal is sent to a daemon process (e.g. 202) that inspects the most recently captured clip and 
determines whether it should be saved. The daemon could be scheduled so that it wakes up N 
hours after the clip is captured. This allows a user to see everything that happened, even if no 
motion was detected. The monitoring application 202 can incorporate a decay factor (which the 
scheduling software 220 can track) so that initially it keeps everything. Then it deletes clips with 
no motion at all. And as time goes on it deletes clips with greater and greater motion. 
Eventually, the clips left show the main activities during that time period, for example, two 
people standing around the printer fighting over what printout they're supposed to get. In a 
desktop printer, initially this would delete clips that show no one in the office. Then it would 
delete clips that show the user sitting at a PC typing (small motion), then it would delete the user 
moving to pick up the phone. Eventually the clips left show visitors walking in and out of an 
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office. This technology could be used as a burglar-monitoring tool. This system could output 
video paper that shows key frames with significant events detected by this system and bar codes 
that point to the video clips. 

[0056] In another example, the printer can listen to what happens around it. The 
printer 104 has an audio capture interface (e.g. microphone 168 or connector for a microphone 
136, and an audio card 136.) The data is recorded on the printer and time stamped so it can be 
retrieved by time from the printer 100, for example from the internal web server 1 10 in the 
printer 100. In a workgroup setting, this function can record conversations around the printer. It 
can also record sounds like that made by someone grabbing a printout from the output tray. In a 
desktop printer, this capability would let the printer monitor conversations in an office between 
the occupant and visitors or between the occupant and people he talks with on the phone. It 
could also be programmed to recognize key clicks, phone rings, and the noise made by the 
printer when it generates paper. The printer could generate an audio or video paper printout that 
summarizes the audio it captures. 

[0057] In another example, the printer 100 can monitor the phone numbers called 
and received as well as record the telephone calls themselves. The printer 100 can be equipped 
with a RJ-1 1 pass- through 136 (2 connectors - one for line-in and the other to the desktop 
phone.) - This allows the printer to tap a phone call, record numbers called, and numbers 
supplied by caller-id. If audio capture hardware (e.g. microphone 168 and audio to digital 
converter 136) is also present, this capability lets the printer record phone calls. The printer 
could provide the beep tone that notifies callers they are being recorded. In any case, this printer 
100 could produce a phone log showing the times when calls were placed, their duration, the 
phone numbers, and bar codes that point to the recorded conversations. 
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[0058] Of course, any of these PC interaction monitoring functions can be 
combined and their data correlated on a content index document such as a video or audio paper 
document. 

[0059] Figure 6 illustrates a method for editing time based data content for use in a 
printer with an embedded multimedia server in accordance with an embodiment of the present 
invention. The multimedia content editing software portion 218 selects 618 portions of content 
based on criteria and automatically generates 620 an index document that is based on the 
selected portions. 

[0060] Figures 7 and 8 illustrate examples of methods in accordance with 
embodiments of the present invention, each of which can be particularly useful for analyzing 
recorded meetings. In one example, an application 202 may include instructions controlling the 
implementation of one or more of these embodiments. 

[0061] Figure 7 illustrates an example of a method for printing a video content 
selection based on user defined criteria wherein the criteria is that each clip indicates a time 
period in which a different person begins speaking for use in a printer with an embedded 
multimedia server in accordance with an embodiment of the present invention. 

[0062] Consider an example in which the printer 100 receives time based content 
data of a video recorded using a 360 degree meeting recorder plus multi-channel audio inputs. 
The control unit 214 retrieves 602 the video recorded with the 360 degree input and a multi- 
channel audio input. A meeting recorder application 202 invokes the multimedia content 
recognition 228 which applies 604 speech recognition method(s) to the audio input to determine 
which participant is the speaker. Additionally, either the meeting recorder application 202 or the 
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recognition software 228 can apply 606 sound localization method(s) based on the audio input 
and the 360 degree input to determine which participant is the speaker. 

[0063] Based on the timeline of the video, and the results from the speech 
recognition and sound localization methods, the meeting application 202 invokes the content 
editing software 218 to segment 608 the video into clip(s) in which each clip illustrates a time 
period when a different person begins speaking. The content indexing software portion 208 
generates 620 an index document that shows a key frame every time a different person begins 
speaking as well as the printed speech recognition results. 

[0064] Figure 8 illustrates an example of a method for printing a video content 
selection based on user defined criteria wherein the criteria is that each clip indicates a time 
period in which a person was speaking in the location for use in a printer with an embedded 
multimedia server in accordance with an embodiment of the present invention. Consider another 
example in that the meeting video includes a separate multi-channel audio track that was 
produced from a known configuration of microphones designed to optimize audio source 
localization. The control unit 214 retrieves 702 video including a separate multi-channel audio 
track that was produced from a configuration designed to optimize audio source localization. 
The user interface portion 210 receives 704 user input indicating location of interest from a user 
input device embedded in the chassis of the printer. Based on the user input, either the meeting 
recorder application 202 or the recognition software 228 can apply 706 sound localization 
method(s) based on the audio track to determine when a person in the location of interest was 
speaking. Based on the timeline of the video, and the sound localization results, the meeting 
application 202 invokes the content editing software 218 to segment 708 the video into clips(s) 
in which each clip illustrates a time period when a person was speaking in the location. The 
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content indexing software portion 208 generates 620 an index document illustrating key frames 
which can also include bar codes for the selected clips. 

[0065] Figure 9 illustrates an example of a method for printing a video content 
selection based on user defined criteria wherein the criteria is that each clip indicates a time 
period in which a participant is present in a video selection for use in a printer with an embedded 
multimedia server in accordance with an embodiment of the present invention. Consider another 
example in which the printer edits time based content for identifying and cataloging the presence 
of an entity such as a participant in the content. This method can be embodied in an entity 
identification application 202. The control unit 214 retrieves 802 a video from storage and 
retrieves 804 user input indicating a participant whose presence in the video is to be indexed. 
The entity identification application 202or the recognition software 228 performs 806 face 
recognition method(s) on each frame to identify a visual appearance of a participant. The 
recognition software 228 performs 808 voice matching method(s) on each frame to identify 
frames in which the participant speaks. Based on the timeline of the video, and the results from 
the face recognition and voice matching methods, the entity identification application 202 
invokes the content editing software 218 to segment 810 the video into clip(s) in which each clip 
illustrates a time period when the participant is speaking or is seen in the video. The content 
indexing software portion 208 generates 620 an index document illustrating or referencing the 
selected clips indicating the participant's presence. Printouts can be generated that list every 
appearance of a given participant. An example application would be "printing" a TV program 
and generating a list of every appearance of a given actor. 

[0066] Figure 10 illustrates an example of a method for printing a video content 
selection based on user defined criteria wherein the criteria is that each clip indicates a time 
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period in which a person was speaking in a video for use in a printer with an embedded 
multimedia server in accordance with an embodiment of the present invention. Consider another 
example in which the printer automatically segments time-based media into clips based on 
events in an audio channel. For example, every time someone speaks, the system constructs a 
clips that spans the time from 30 seconds before the person started speaking until 30 seconds 
after they stopped speaking. This would be useful for "print" surveillance videos where there are 
long periods where no one says anything and brief times when someone in the field of view says 
something. This method can be embodied in an event detection application 202 The control unit 
214 retrieves 902 video including an audio channel from storage. Based on the timeline of the 
audio channel, and the detection of an audio event, the event detection application 202 invokes 
the content editing software 218 to segment 904 the video into clip(s) in which each clip 
illustrates a time period from 30 seconds before a person starts speaking or another audio event 
occurs until 30 seconds after that person has stopped speaking or the event has ended. This 
method can of course be applied to detection of other events such as changes in the field of view 
of an image capture device. 

[0067] As meetings are often based around presentations, a method for editing 
content from a captured presentation for use in a printer with an embedded multimedia server in 
accordance with another embodiment of the present invention is complementary and of course 
can also be used independently. This method can be embodied in a presentation capture 
application 202 in the embodiment of Figure 2. Consider the example in which the printer 100 
has VGA-to-NTSC conversion hardware 136 and audio capture capabilities 168, 136. A user 
plugs a laptop into one side of the printer and plugs a projector into the other. The printer 100 
captures video frames as fast as presentation capture software 202 compares or has them 
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compared to the most recently captured frame, and retains those frames that are different. A 
parallel audio track is also saved. This presentation capture capability could also be used in a 
desktop printer to record a presentation made on a PC connected to the printer. For example, 
someone practicing a presentation would be a potential user. It could also be used to record 
activities on a PC during a critical time, e.g., while developing code. Off-loading these functions 
to the printer significantly reduces the computational load on the PC. The content indexing 
software 208 creates a video paper representation that shows the frames and/or bar codes that 
allow the user to replay the audio associated with the frames. The printer can serve the audio 
itself or it can be written to a digital medium like an SD disk that can be played from a cell 
phone or a PDA. The audio could also be written to a 2D bar code. Packaging these functions in 
the printer provides a convenient and easy-to-use a presentation recorder. 

[0068] In another version, the printer receives video of people nearby a printer. In 
this example, a user is giving a presentation near the printer or working on his or her PC. This 
adds significant information to the recording. The data is saved on the printer. Key frames are 
chosen from the video stream and added to the video paper printout. 

[0069] Figure 1 1 illustrates an example of a method for capturing referenced 
multimedia content in a content selection designated for printing for use in a printer with an 
embedded multimedia server in accordance with an embodiment of the present invention. The 
illustrative example in this context is that the printer automatically extracts URL's from the 
documents (e.g. embodied as postscript files) it receives for printing. In the embodiment of 
Figure 1, the web server 1 10 fetches the referenced content at the URL and constructs a printable 
representation, for example converting HTML to PDF. The multimedia server 104 constructs 
thumbnail images for the referenced content, for example WebPages, suitable or specialized for 
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display on the printer's display 148 perhaps with Ricoh's SmartNails technology. The display 
148 presents the user with a list of the documents that the printer 100 has identified are in its 
output tray. The user chooses his document and is presented with a list of pre-fetched URL 
references that can be printed on the spot because they've been pre-fetched. 

[0070] The multimedia server 104 receives events 1002 and directs them to the 
appropriate software portion for performing an associated action 1004. The multimedia server 
104 receives 1006 a document identified for printing. The server searches the document for 
references to online content. URL's are examples of references used for this example. The 
server 104 extracts 1008 each URL from the document, and the web server 1 10 retrieves 1010 
content from each URL. The multimedia server 104 constructs 1012a printable representation 
of the retrieved content for each URL and associates 1014 each printable representation with a 
respective display identifier suitable for display on the embedded printer display 148. 
Responsive to the event that the document associated with a set of retrieved content has been 
selected 1016, the multimedia server 104 displays 1018 the display identifier(s) associated with 
the document on the embedded printer display 148. Responsive to an associated display 
identifier having been selected 1020, the multimedia server 104 directs the print engine 140 to 
print 1022 a printable representation of the retrieved content on an indicated medium. 
Responsive to an indication that the associated document is no longer available 1024, the 
multimedia server 104 updates the display 148 to remove 1026 the document from the displayed 
list of documents in the output tray. 

[0071] Figure 12 illustrates an example of a method for determining whether a 
printed content selection has been removed from an output holder for use in a printer with an 
embedded multimedia server in accordance with an embodiment of the present invention. In this 
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example, the estimation of the documents in the tray and the documents pulled out of the tray is 
done by weight, as the output tray in this embodiment includes a scale [not shown] that can 
compute the weight of the sheets of paper it holds. From this it estimates the number of sheets in 
the output tray. It knows when a user withdraws paper a change in weight. The document 
choices presented the arriving user can expire after N hours. When the user withdraws a 
document but leaves others in the tray, the list of choices is modified so that the best guess is 
highlighted or moved to the top of the list. For example, if one user prints a ten page document 
and another user prints a five page document, the output tray would weigh 15 pages. When the 5 
page document is withdrawn it would weigh 10 pages thereby allowing the printer to estimate 
that the URL's in the 5-page document should be displayed. 

[0072] Again, the multimedia server 104 receives events 1002 and directs them to 
the appropriate software portion for performing an associated action 1004. Responsive to 
receiving a request for Calibration 1 106, the multimedia server 104 calibrates 1 108 or directs the 
print engine 140 to calibrate 1 108 the scale of output tray to the weight of a number of units of 
output media. Responsive, to receiving 1 1 10 an indicator of a document beginning printing to 
the output tray, the print engine 140 determines 1 1 12 the weight change in the output tray, and 
associates 1 1 14 the weight change as the weight of the document. The multimedia server is 
notified that the associated document is in the output tray, and it sets 1116a timer for a timeout 
window for pickup of the document. Responsive to receiving 1 1 18 an indicator of decrease in 
weight of the output media in the output tray, software executed by either the print engine 140 or 
the server 104 determines 1 120 which document identified as in the tray has the closest weight, 
and the multimedia server 104 displays 1 122 the display identifier(s) of retrieved content from 
any URLs in the determined document. Responsive to the timeout window elapsing 1 124, the 
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multimedia server removes 1 126 the document from displayed list of documents in the output 
tray. 

[0073] In another embodiment, the printer 100 provides an embedded text-to- 
speech system. This provides an aid for the visually impaired, for example a TTS capability. 
The user interface 152 can include a button requesting that a text document be read in an audio 
form. In another embodiment, the printer 100 includes a microphone 168 for receiving speech 
commands requesting the reading of a document in audio form. Text to speech software can 
generate an audio version of the document which is transmitted to the speaker 150. A user who 
prints a text file or web page can walk up to a printer and listen to the contents of the file while it 
is being printed or previewed. The output of this system could also be previewed in a print 
dialog box. In another example, the multimedia server 104 delivers the audio version to a 
networked external computer 154. 

[0074] In another embodiment, the text to speech software can read a summary of a 
document either while standing in front of the printer or at a networked computer before 
deciding to generate a paper printout. The output of this system could also be previewed in a 
print dialog box. 

[0075] The foregoing description of the embodiments of the invention has been 
presented for the purpose of illustration; it is not intended to be exhaustive or to limit the 
invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate that 
many modifications and variations are possible in light of the above teachings. It is therefore 
intended that the scope of the invention be limited not by this detailed description, but rather by 
the claims appended hereto. 
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